After encountering some memory errors while parsing the data downloaded from Mapzen, I converted the original file to a smaller sample, using the code provided at the instructor note. File downsized from 300Mb to 30Mb. Later on, after having changed the pretty event from "True" to "False" and parsing on the original data file, I encountered one error and one restriction. The error was the fact that some street names had multiple abbreviations and the restriction that the postalcodes should have only 4 digits in order to be valid.
Before importing the data to MongoDB, I updated all substrings in problematic address strings. The code used and the output strings corrected were the following:
import xml.etree.cElementTree as ET
from collections import defaultdict
import re
import pprint
OSMFILE = "C:/Users/oikonomakisa/Desktop/wellington_sample.osm"
street_type_re = re.compile(r'\b\S+\.?$', re.IGNORECASE)
expected = ["Street", "Avenue", "Boulevard", "Parade", "Wakefield/Cuba", "Place", "Square", "Lane", "Road", "Terrace", "Crescent", "Way", "Grove",
"Trail", "Parkway", "Commons", "Drive", "Esplanade", "Quebec", "South"]
mapping = {"St": "Street",
"street": "Street",
"avenue": "Avenue",
}
def audit_street_type(street_types, street_name):
m = street_type_re.search(street_name)
if m:
street_type = m.group()
if street_type not in expected:
street_types[street_type].add(street_name)
def is_street_name(elem):
return (elem.attrib['k'] == "addr:street")
def audit(osmfile):
osm_file = open(osmfile, "r")
street_types = defaultdict(set)
for event, elem in ET.iterparse(osm_file, events=("start",)):
if elem.tag == "node" or elem.tag == "way":
for tag in elem.iter("tag"):
if is_street_name(tag):
audit_street_type(street_types, tag.attrib['v'])
return street_types
def update_name(name, mapping):
m = street_type_re.search(name)
if m:
street_type = m.group()
if street_type not in expected:
name = re.sub(street_type_re, mapping[street_type], name)
# pprint.pprint(name)
return name
def test():
st_types = audit(OSMFILE)
for st_type, ways in st_types.iteritems():
for name in ways:
better_name = update_name(name, mapping)
print name, "=>", better_name
test()
152 Riddiford street => 152 Riddiford Street
Moxham avenue => Moxham Avenue
Willis St => Willis Street
The issue has been managed by augmenting the code with an exra rule and calling the above function at the Final Project:
def postcode_checker(v):
"""
Checks postcodes and reduces to 4 digit strings.
"""
postcode = ''
for char in v:
if char.isdigit():
postcode += char
if len(postcode) == 4:
break
return postcode
This section contains basic statistics about the dataset and the MongoDB queries used to gather them.
wellington_new-zealand.osm............335Mb
wellington_sample.osm....................33.8Mb
wellington_sample.osm.json............34.8Mb
db.Project.find().count()
168203
db.Project.find({"type":"node"}).count()158653
db.Project.find({"type":"way"}).count()9550
db.Project.distinct("created.user").length327
db.Project.aggregate([{"$group":{"_id":"$created.user", "count":{"$sum":1}}}, {"$sort":{"count":-1}}, {"$limit":1}])
{"id_" : "LINZ Data Upload", "count" : 48506}
db.Project.aggregate([{"$group":{"_id":"$created.user", "count":{"$sum":1}}}, {"$group":{"_id":"$count", "num_users":{"$sum":1}}}, {"$sort":{"_id":1}}, {"$limit":1}])
{"_id" : 1, "num_users" : 93}
Actually the osm data of Wellington, New-Zealand was pretty clean and structured. I thought of putting an extra boolean field of cycleway in pedestrian-only streets but it was already highlighted(k: cycleway, v: yes) in specific roads. So the rest I can do is to make some extra queries on the data.
db.Project.aggregate([{"$match":{"amenity":{"$exists":1}}}, {"$group":{"_id":"$amenity",
"count":{"$sum":1}}}, {"$sort":{"count":-1}}, {"$limit":10}])
{"_id" : "parking", "count" : 105}
{"_id" : "school", "count" : 27}
{"_id" : "bench", "count" : 22}
{"_id" : "cafe", "count" : 18}
{"_id" : "fast_food", "count" : 17}
{"_id" : "restaurant", "count" : 16}
{"_id" : "toilets", "count" : 14}
{"_id" : "place_of_worship", "count" : 13}
{"_id" : "post_box", "count" : 11}
{"_id" : "fuel", "count" : 11}
db.Project.aggregate([{"$match":{"amenity":{"$exists":1}, "amenity":"place_of_worship"}},{"$group":{"_id":"$religion", "count":{"$sum":1}}},{"$sort":{"count":-1}}, {"$limit":2}])
{"_id" : "christian", "count" : 12}
{"_id" : "muslim", "count" : 1}
db.Project.aggregate([{"$match":{"amenity":{"$ne":"NULL"}, "amenity":"restaurant"}}, {"$group":{"_id":"$cuisine", "count":{"$sum":1}}},{"$sort":{"count":-1}}, {"$limit":5}])
{"_id" : "null", "count" : 11}
{"_id" :"japanese", "count" : 1}
{"_id" : "malaysian", "count" : 1}
{"_id" : "indian", "count" : 1}
{"_id" : "mexican", "count" : 1}
One can observe that the most popular cuisine is marked with null value. It could probably be the local cuisine and could be changed to "local".
The most accurate thing to implement in our case is gamification in data entry. There are many benefits associated with gamification. Ensuring data quality has become one of the most important with the advent of “Big Data” decision making. By introducing gamification to the work environment, managers create an engaging user experience and most importantly increase the quantity and quality of data for decision making. Why not doing something similar to the OpenStreetMap.org project? Why each member who can import data to the platform, shouldn't have a profile with a rating level. The rating should have two steps. One for the quantity of data imported(or changed) and one for the quality of data from an authorized reviewer(or a super user who should review other users once each week if he wants to keep being in the hall of fame). By doing so, users are competing against each other for rating level. Status is clearly visible, so even if users are not competing against each other, there is still incentive to achieve.